Repository navigation
feat(telemetry): the numbers a spike needs are exported, alerted on, and reach the public health check (Scaling H5) - #2584
Conversation
…and reach the public health check (#2541) Four series (viewers per world, the launcher's own loop delay p99, worlds created, disk free/size), four rules (launcher-loop-stall, worlds-created, disk-free, and viewers-high as information only), and a problems file the collector rewrites after each scrape: every firing rule that pages is an alert:<rule> host code in /__sim/healthz at once (hostOk false), cleared when it resolves; a dead collector is alert:stale. sim-worlds-telemetry@.service is installed by install.sh and never enabled. A failed hourly backup (timeout or a FAILED database) raises backup-hourly:<stage> and shows on the health page, like the nightly one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Proof check:
|
| Check | Status | Detail |
|---|---|---|
| problem statement | ok | present |
| linked issue | ok | #2541 |
| acceptance | ok | 3/3 met |
| carried from the issue | ok | all 3 issue criteria present |
| self-audit | ok | checks, failed attempts, long tail and limits filled |
Acceptance criteria
| Criterion | Status | Evidence | |
|---|---|---|---|
| [x] | A1 Tests for each series and each rule (fire and resolve) | met | node --test tools/world-launcher/telemetry-spike.test.mjs -> 7 pass, 0 fail. Covered: the four series and their absenc |
| [x] | A2 Test: a firing rule makes /__sim/healthz report hostOk: false with |
met | node --test tools/world-launcher/telemetry-spike.test.mjs "A2: a firing rule makes /__sim/healthz report hostOk false |
| [x] | A3 docs/operations/telemetry.md documents the series, rules, the unit a |
met | docs/operations/telemetry.md sections "The series a spike needs", "Rules" (four new rows and which rules page), "Reach |
Proof links: 0 video, 0 image, no proof.json.
Updated on every edit, push, label and review. How to fill it: .claude/skills/journal/SKILL.md. T3-T4 also need the maintainer's sign-off (tools/proof/gate.mjs). A maintainer can waive with the proof:waived label.
|
Merge queue: this PR left the queue without merging (reason: Fix it and add |
# Conflicts: # tools/world-launcher/launcher.mjs
Problem
During a spike nobody was told anything. Viewers per world, the launcher's own event-loop delay, worlds created and free disk were not exported. No unit ran the collector with alerts, and an alert that fired stayed a local line in
alerts.jsonl. Closes #2541.Acceptance
node --test tools/world-launcher/telemetry-spike.test.mjs-> 7 pass, 0 fail. Covered: the four series and their absence when unreported.launcher-loop-stallfires within 60 s and resolves below 0.15 s.worlds-createdfires on 11 creations in an hour across a launcher restart, does not fire on 10, and resolves when the hour has passed.disk-freefires and resolves.viewers-highfires and resolves inalerts.jsonland never pages./__sim/healthzreporthostOk: falsewith the code, and resolving clears it — evidence:node --test tools/world-launcher/telemetry-spike.test.mjs"A2: a firing rule makes /__sim/healthz report hostOk false with its code; resolving clears it at once; the launcher exports the series" -> pass. It runs a real launcher: the collector's own evaluation firesdisk-free, healthz sayshostOk: false, hostCodes: ["alert:disk-free"], and after the resolvehostOkis no longer false andhostCodesis gone. A stale timer copy does not hold the code, and a dead collector shows asalert:stale.docs/operations/telemetry.mddocuments the series, rules, the unit and how to enable it — evidence:docs/operations/telemetry.mdsections "The series a spike needs", "Rules" (four new rows and which rules page), "Reaching a person" and "The unit" (enable, check and stop commands)Proof
Tier: T1 (
node tools/proof/tier.mjs --base origin/main-> T1)Evaluation cost with the new rules (the issue asked for it again). The run used 400 registered worlds (20 running) and one synthetic hour: 240 scrapes, alerts evaluated after each. Base (the rules on main) and candidate ran back to back, two repeats each, on a shared 16-core workstation with a 1-min load of 5.6–7.6. These are in-process timings of one function, not a load run:
That is about 2 ms more per scrape, still about 0.3 % of one core at the 15 s cadence.
Self-audit
node --test tools/world-launcher/telemetry-spike.test.mjs tools/world-launcher/telemetry-alerts.test.mjs tools/world-launcher/telemetry.test.mjs tools/world-launcher/metrics.test.mjs tools/world-launcher/host-health.test.mjs tools/world-launcher/health-dash.test.mjs tools/world-launcher/health-dash-backup.test.mjs tools/world-launcher/health-dash-auth.test.mjs tools/world-launcher/telemetry-cost.test.mjs tools/world-launcher/telemetry-s2.test.mjs tools/world-launcher/telemetry-units.test.mjs tools/world-launcher/ceilings.test.mjs tools/world-launcher/guard.test.mjs tools/world-launcher/account-creation.test.mjs tools/world-launcher/heap.test.mjs tools/world-launcher/ship-root.test.mjs-> 126 pass, 0 fail, 0 skippednode --test tools/world-launcher/keeper.test.mjs tools/world-launcher/launcher.test.mjs tools/world-launcher/shed.test.mjs-> pass (after the fix under Failed attempts)bash -n tools/world-launcher/deploy/install.sh-> oknode .github/scripts/guards.mjs scrub --base origin/main-> 0 problemsdeploy/host-health.mjsand had the launcher import it from there. The keeper test's release copy leaves outdeploy/, so the keeper's launcher could not start (7 failures inkeeper.test.mjs). The reader moved totelemetry/alerts.mjs, beside its writer, and host-health imports it.telemetry-alerts.test.mjs) used a launcher fixture without the new series, so three new fleet rules wentunknownin it. The fixture now carries them, quiet, as a launcher of this release does.statfsand a histogram read in the launcher. Per healthz, one small file read (absent until the unit is enabled). About 2 ms more evaluation per collector scrape. No world is woken; no Soul or account data is touched. Labels are the world slug only.sim_world_viewersreads the launcher'slivemap as it is on main; if V6 changes where viewers are counted, that map is the one place to keep in step.worlds-createdcounts only rises the collector saw: creations before its first scrape are not counted.refusals(see Deviations), so a world overtick-sloworgc-pressuremakeshostOkfalse, as the design says. The release snapshot needshostOk, so a release during a firing alert needs--accept-host-codes.docs/operations/telemetry.md(series, rules, reaching a person, the unit, cost).Deviations from the design
sim_host_disk_size_bytessits besidesim_host_disk_free_bytes, because "under 20 % free" needs the total.refusalsdoes not page, likeviewers-high. It fires at once on any single refused request, which is the ceilings doing their job. Paging on it would make the public health red on every rate limit.<data>/alert-problems.json, written by the collector. The launcher's healthz reads it on every request, the way it reads the guard's world problems.deploy/host-health.mjsfolds it every 10 min, which also logs ALERT lines. Healthz trusts its own read over the timer's copy, so a resolve clears at once. A dead collector (file older than 4 scrapes and 2 min) isalert:stale; a clean stop removes the file.SIM_WORLDS_ARGS.collect --launcher-argsreads the port and base from/etc/sim-worlds/<stage>.conf, so the unit carries no port.sim-worlds-backup-hourly@<stage>, from the hourly-backup PR not yet merged) raisesbackup-hourly:<stage>in host-health, like the nightlybackup:<stage>. The health page alerts on it: a FAILED database frombackup-hourly-last.json, or the unit's result for a timeout, which writes no record.Outside Touches
tools/world-launcher/health-dash.mjs: 5 lines, for the hourly backup on the health page (the coordinator's addition).Time
Started 07:57 UTC, PR at 08:17 UTC; ceiling 3 h.
Changelog
Labelled
no-changelog: operator tooling (metrics, alerts, a unit, the health page); nothing changes for people using Souls.Risk
Low until the unit is enabled: without
alert-problems.json, healthz is unchanged. The launcher's additions are one histogram and onestatfsper scrape. Once the unit is enabled, a firing alert makes the publichostOkfalse. That is the point, but a noisy rule would page; Arules.jsonoverride (--rules, added to the unit by a drop-in) can raise a threshold or turn a rule off without a release. To roll back, disable the unit (which removes the file) or revert the commit.🤖 Generated with Claude Code